Skills Lab 02: Asking Good Qs

.title[
# Skills Lab 02: Asking Good Qs
]
.subtitle[
## <i>Do you really Likert, is it is it wicked</i>
]
.author[
### Dr Danielle Evans
]
.date[
### 2 February 2023
]

---

## Session Overview

- **Designing Effective Questionnaires**

+ From ideas to items
  
  + Question & response types
  
  + Example items

- **Designing Effective Interviews**

+ From ideas to interviews
  
  + Interview schedules

+ Successful probes & follow-ups

+ Piloting

---

## Part 1: Quantitative | Questionnaires

---

## Introduction

- In psychology, we often want to measure things we cannot **directly observe**

- Within **quantitative methods**, measurement refers to the way we record behaviours, feelings, attitudes, & characteristics in a **quantifiable sense**

- We can measure variables like height & weight, but we can also measure **'unquantifiable' constructs** like 'happiness' or 'lecture-induced boredom'

- We can make attempts to measure these feelings through **observation**...

+ I could count how many times you smile during a conversation, or how many times you yawn in the next hour

- But often we rely on our participants' abilities to **self-report** their behaviours, feelings, attitudes, & characteristics instead

<br>

<div class="sweat" style="font-size:90%">
<p><b>But!</b> We need to make sure that we ask our participants the <b>right questions</b> so that they can <b>self-report accurately...</b> this applies to quant & qual methods! </p>
</div>

---

## From Ideas to Items

- We most often ask participants to self-report through **questionnaires**, BUT someone has to create them at some point... (your task for the **lab report**)

- When designing any questionnaire, the first task is identifying and **defining** what topic(s) we want to be covered in the survey

- The aim here is to break down **big ideas** into **smaller concepts** that can be measured with **specific items** in a questionnaire

- For our session today, let's say that our topic is '**lecture-induced boredom**' 😴

+ With the rationale that in the future I can administer this questionnaire after every class to see what teaching techniques and subjects are the least boring...

- The next step is to brainstorm what **thoughts**, **feelings**, **behaviours**, or **characteristics** define our construct

<div class="pc" style="font-size:90%">
<p><b>Task!</b> It's Poll Everywhere time! Get your devices out ready! </p>
</div>

---

## From Ideas to Items: Define

#### So, what even is 'lecture-induced boredom'?

<div class="tu" style="font-size:90%">
<p><b>Top Tip!</b> Picture people at the two extremes of lecture-induced boredom, what do they 'look' like? What do they think/feel? What behaviours do they display? </p>
</div>

---

## Question Types

- So now we have an idea of what 'lecture-induced boredom' is, we can design some questions to measure it...

- For our next step, we need to decide what **type** of questions we're going to ask:

- **Open Questions**
  + Empty text boxes

- **Closed Questions**

+ Yes/no
  + Multiple choice
  + Ranking scales
  + Likert scales

---

## Question Types

- So now we have an idea of what 'lecture-induced boredom' is, we can design some questions to measure it...

- For our next step, we need to decide what **type** of questions we're going to ask:

- **Open Questions**
  + Empty text boxes

- **Closed Questions**

+ Yes/no
  + Multiple choice
  + Ranking scales
  + **Likert scales**

<br>

### *That's the way, aha, aha, I Likert*

]

---

## Likert Scales

- Usually 5 or 7 points covering the two extremes with a neutral option in the middle e.g.,

+ **Not very often** to **very often**
  
  + **Very untrue of me** to **very true of me**  
  
  + **Not at all important** to **extremely important**
  
  + **Extremely unlikely** to **extremely likely** 
  
  + **Strongly disagree** to **strongly agree**
  
  + **Never true** to **always true**
  
<br>

<div class="sweat" style="font-size:90%">
<p><b>Easy Error!</b> The responses must fit with the phrasing of the question or statement! For today, let's go with <b>strongly disagree</b> to <b>strongly agree</b> </p>
</div>

---

## From Ideas to Items

- Now we've decided on a **Likert scale** we can come up with our **items**!

+ **Items** in this context, refers to the different **statements** we want to include in our questionnaire
  
- As a starting point, it can be helpful to look at how our construct has been measured previously, or how it's been measured in similar contexts

- We can use those examples as inspiration for coming up with our own items (but don't steal!!)

- Looking at existing measures can also help us identify what gaps there are in previous work that our questionnaire can fill

<div class="mb" style="font-size:90%">
<p><b>Keep in Mind!</b> We want our items to be <b>valid</b> and <b>reliable</b> measures of the specific construct we're interested in! 🎯  </p>
</div>

---

## My Attempts

- I had a go at coming up with my own items in the next few slides...

- I used the [Boredom Proneness Scale](https://www.tandfonline.com/doi/pdf/10.1207/s15327752jpa5001_2?needAccess=true) by Farmer & Sunberg (1986) as inspiration...

- Using what you already know about questionnaire design, I want you to have a go at telling me what's right or wrong with them...

---

## My Attempts

#### 1. What do you think about this lecture?

]

<div class="pc" style="font-size:90%">
<p><b>Task!</b> What do you think?  Post on poll everywhere! </p>
</div>

---

## My Attempts

#### 2. Unless I am doing something exciting‚ even dangerous‚ I feel half-dead & dull.

]

---

## My Attempts

#### 3. Many lectures I attend are repetitive and monotonous.

]

---

## My Attempts

#### 4. How bored have you been in your lectures in the past month?

]

---

## My Attempts

#### 5. It is not easy for me to not concentrate on my educational activities.

]

---

## My Attempts

#### 6. I find my lectures boring.

]

---

## My Attempts

#### *Select how much you agree with the following statement.*
#### 7. I feel inspired after attending this lecture.

]

---

## Some Final Q-Tips

- Use **statements** not questions

- Make sure your response options fit the wording of the item

- Be specific, ask one thing per question

- Avoid **double-negatives** & **double-barrelled** questions

- Use **neutral language** - don't present one response as being more desirable

- Avoid **jargon**

- Consider your **sample**, are there any factors that might impact their understanding of your items?

- **Pilot** your questionnaire to ensure it makes sense to others

<div class="tu" style="font-size:90%">
<p><b>Top Tip!</b> Keep the image of the 'extremes' in mind when creating and refining your own questionnaire items in your practicals! </p>
</div>

---

## Part 2: Qualitative | Interviews

---

## Qualitative Methods

- With **qualitative methods** we are usually trying to understand someone's personal experiences

- We're more concerned with meaning and understanding **subjective experiences**

- There are *many* different types of qualitative research, and a lot of them involve using **interviews** as a method of data collection

- 3 different types of interview: 
  
  + **unstructured** 
  + **semi-structured**
  + **structured** 
  
- Could involve individuals OR groups

<div class="mb" style="font-size:90%">
<p><b>Be Aware!</b> Both quantitative and qualitative methods are <b>subjective</b>, but we'll learn more about that through this module! 😁 </p>
</div>

---

## From Ideas to Interviews

- The process of coming up with questions for interviews is quite similar to questionnaires...

- You firstly need to identify the research topic or area of interest

- For unstructured interviews the research topic can be relatively general, for structured interviews it needs to be more focused

- For more structured interviews, you would then create an **interview schedule** (or an **interview guide**) that contains the questions you plan to ask all of your participant(s)

+ For more **sensitive topics** you might also create a **distress protocol** based on the questions in your interview schedule (we'll talk more about ethics in interviewing next week!)

---

## From Ideas to Interviews

- To help you come up with questions, think about what type of information you would like to obtain: **opinions**, **attitudes**, **knowledge**, **beliefs**, **behaviours**, **sensory**, **experiences**, **feelings**, **demographics**...

- Create a bank of questions that cover each of these domains for your research topic, focusing on the **past**, the **present**, and the **future**

- e.g. "Thinking back to when you were a child, how did you feel about transitioning to high school? (**feelings** in the **past**)

- Aim for open questions that elicit **story-telling** - remember the goal is to understand your participants' experiences in depth

- **Test them out** - think about how you might answer them yourself/go through the questions with a friend

---

## Let's give it a go!

If we were interested in **R anxiety** what interview questions could we ask?

<div class="tu" style="font-size:90%">
<p><b>Keep in mind!!</b> We want to know attitudes, knowledge, beliefs, behaviours, experiences, feelings, demographics etc, in the past, present, & future</p>
</div>

---

## Swaying from the Schedule

- Sometimes you might need to ask questions that are not in your **interview schedule**

- Participants could offer up interesting points that you hadn't originally considered when creating your interview schedule, but now want to explore further

- Or it could be that your participant isn't very forthcoming, or something they've said is inconsistent/unclear

- In these cases, you can sway from the interview schedule to **probe** or **follow-up** on prior responses

---

## Successful Probes & Follow-ups

- Avoid directly asking '**why?**' - it can come off as confrontational, instead, try using '**how**'

- Pay attention to your participant and actively listen

+ Otherwise you might miss important opportunities to probe!

- When you create your interview schedule, include some **answer-based paths** to take to help you prepare follow-ups and probes in advance

+ I.e., if the participant says **x**, then I'll ask them **y**

- Be flexible when creating your interview schedule, sometimes things you don't anticipate might come up

<br>

<div class="tu" style="font-size:90%">
<p><b>Top Tip!</b> Be comfortable with silence and your participant will want to fill it! Give them space and time to think! </p>
</div>

---

## Piloting

- It's easier to clarify what you mean by a question during an interview because you're present and involved in the process (vs a questionnaire...)

- But it's much better to avoid any misunderstanding from the start - participants may feel uncomfortable asking for clarity

- You should **pilot** your interview questions (and questionnaires too) before recruiting participants for the main study

- Some questions might turn out to be **easily misunderstood** or actually **irrelevant** to the research question

- After piloting, you can revise any problematic questions to be more effective

<br>

<div class="mb" style="font-size:90%">
<p><b>Keep in Mind!</b> The process of designing a questionnaire or an interview schedule is very reflective & iterative! It requires testing & 'thinking time' 💭 </p>
</div>

---

## Best Practice

- Ask one question at a time, avoiding questions that elicit short answers

- It's important to build **rapport** so start with simple questions to ease your participant into the interview

+ Choose questions that your participant won't find too difficult to answer, they're likely to feel a little nervous, & might need some time to warm up!

- When moving between topics, tell your participant and give them the opportunity to add anything before you go onto the next question

- Keep difficult or potentially **sensitive questions** towards the end of the interview (when you both feel more comfortable with the interview process)

- Close the interview with an **opportunity for participants to add** any more information ("is there anything else you’d like to add before we end?")

- Tailor the **language** and **pace** of the interview to your **sample**

+ Consider any factors (such as age, culture, language etc.,) that might affect how participants understand and respond to your questions

---

## Summary

- We heavily rely on **self-report measures** in psychology, and so we need to design these measures to be **valid** and **reliable**

+ To get good data, we need to ask good questions!!

- The process of creating questions (for surveys OR interviews) generally starts with **defining** our research topic or area of interest

- We then want to design questions that look at the different **thoughts**, **feelings**, **behaviours**, **characteristics**, **opinions**, **attitudes**, & **experiences** related to our research topic

- We should **pilot** any **questionnaires** or **interview schedules** before administering them to ensure there aren't any errors in understanding, and to make sure our questions are actually relevant to our topic, making any adjustments as necessary

- We can then evaluate how 'good' our questions are with statistics, which we'll be covering in the next few weeks! Woohoo! 😁

---

## *That's all - happy questioning!*

<br>

]

[Give session feedback here!](https://forms.gle/ZyXAB7kZzUUyct9n6)

]
]